Search Results: "igor"

16 June 2022

Dirk Eddelbuettel: RcppArmadillo 0.11.2.0.0 on CRAN: New Upstream

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 991 other packages on CRAN, downloaded over 25 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 476 times according to Google Scholar. This release brings a second upstream fix by Conrad in the release series 11.*. We once again tested this very rigorously via a complete reverse-depedency check (for which results are always logged here). It so happens that CRAN then had a spurious error when re-checking on upload, and it took a fews days to square this as everybody remains busy but the release prepared on June 10 is now on CRAN. The full set of changes (since the last CRAN release 0.11.1.1.0) follows.

Changes in RcppArmadillo version 0.11.2.0.0 (2022-06-10)
  • Upgraded to Armadillo release 11.2 (Classic Roast)
    • faster handling of sparse submatrix column views by norm(), accu(), nonzeros()
    • extended randu() and randn() to allow specification of distribution parameters
    • internal refactoring, leading to faster compilation times

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

15 May 2022

Dirk Eddelbuettel: RcppArmadillo 0.11.1.1.0 on CRAN: Updates

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has syntax deliberately close to Matlab and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 978 other packages on CRAN, downloaded over 24 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 469 times according to Google Scholar. This release brings a first new upstream fix in the new release series 11.*. In particular, treatment of ill-conditioned matrices is further strengthened. We once again tested this very rigorously via three different RC releases each of which got a full reverse-dependencies run (for which results are always logged here). A minor issue with old g++ compilers was found once 11.1.0 was tagged to this upstream release is now 11.1.1. Also fixed is an OpenMP setup issue where Justin Silverman noticed that we did not propagate the -fopenmp setting correctly. The full set of changes (since the last CRAN release 0.11.0.0.0) follows.

Changes in RcppArmadillo version 0.11.1.1.0 (2022-05-15)
  • Upgraded to Armadillo release 11.1.1 (Angry Kitchen Appliance)
    • added inv_opts::no_ugly option to inv() and inv_sympd() to disallow inverses of poorly conditioned matrices
    • more efficient handling of rank-deficient matrices via inv_opts::allow_approx option in inv() and inv_sympd()
    • better detection of rank deficient matrices by solve()
    • faster handling of symmetric and diagonal matrices by cond()
  • The configure script again propagates the'found' case again, thanks to Justin Silverman for the heads-up and suggested fix (Dirk and Justin in #376 and #377 fixing #375).

Changes in RcppArmadillo version 0.11.0.1.0 (2022-04-14)
  • Upgraded to Armadillo release 11.0.1 (Creme Brulee)
    • fix miscompilation of inv() and inv_sympd() functions when using inv_opts::allow_approx and inv_opts::tiny options

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

5 April 2022

Dirk Eddelbuettel: RcppArmadillo 0.11.0.0.0 on CRAN: Upstream Updates

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 972 other packages on CRAN, downloaded over 24 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 465 times according to Google Scholar. This release brings a new upstream release 11.0.0. We tested this very rigorously via three different RC release each of which got a full reverse-dependencies run (for which results are always logged here). The full set of changes (since the last CRAN release 0.10.8.1.0) follows.

Changes in RcppArmadillo version 0.11.0.0 (2022-04-04)
  • Upgraded to Armadillo release 11.0.0 (Creme Brulee)
    • added variants of inv() and inv_sympd() that provide rcond (reciprocal condition number)
    • expanded inv() and inv_sympd() with options inv_opts::tiny and inv_opts::allow_approx
    • stricter handling of singular matrices by inv() and inv_sympd()
    • stricter handling of non-sympd matrices by inv() and inv_sympd()
    • stricter handling of non-finitie matrices by pinv()
    • more robust handling of rank deficient matrices by solve()
    • faster handling of diagonal matrices by rcond()
    • changed eigs_sym() and eigs_gen() to use higher quality RNG
    • quantile() and median() will now throw an exception if given matrices/vectors have NaN elements
    • workaround for yet another bug in Intel MKL
  • Until May 2022, protect correction to Field behavior via define of RCPP_ARMADILLO_FIX_Field
  • If a LAPACK installation with missing complex routines is found (as e.g. Ubuntu using 3.9.0) then the LAPACK unit test is skipped.

Changes in RcppArmadillo version 0.10.8.2.0 (2022-02-01)
  • Upgraded to Armadillo release 10.8.2 (Realm Raider)
    • fix potential speed regression in pinv() and rank()

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

6 March 2022

Dirk Eddelbuettel: nanotime 0.3.6 on CRAN: Updates

Leonardo and I are pleased to another update to our nanotime package bringing it to version 0.3.6 which landed on CRAN earlier today. nanotime relies on the RcppCCTZ package (as well as the RcppDate package for additional C++ operations) and offers efficient high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from a rigorous refactoring by Leonardo who not only rejigged nanotime internals in S4 but also added new S4 types for periods, intervals and durations. This release corrects subsetting with %in% operator, integrates it better fit in the mixed S3/S4 setup, fixes a negative period parse, and updates class comparisons to rely on inherits(). The NEWS snippet has the full more details.

Changes in version 0.3.6 (2022-03-06)
  • Fix incorrect subsetting with operator %in% (Leonardo in #100 fixing #99).
  • Fix incorrect parsing for negative nanoperiod (Leonardo in #100 fixing #96).
  • Test for class via inherits() (Dirk).

Thanks to my CRANberries there is also a diff to the previous version. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository and all documentation is provided at the nanotime documentation site. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

15 December 2021

Dirk Eddelbuettel: nanotime 0.3.5 on CRAN: Update

Another (minor) nanotime release, now at version 0.3.5, just arrived at CRAN. It follows the updates RDieHarder 0.2.3 and RcppCCTZ 0.2.10 earlier today in bringing a patch kindly prepared by Tomas Kalibera for the upcoming (and very useful) UCRT changes for Windows involving small build changes for the updated Windows toolchain. nanotime relies on the RcppCCTZ package for (efficient) high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from a rigorous refactoring by Leonardo who not only rejigged nanotime internals in S4 but also added new S4 types for periods, intervals and durations. The NEWS snippet adds more details.

Changes in version 0.3.5 (2021-12-14)
  • Applied patch by Tomas Kalibera for Windows UCRT under the upcoming R 4.2.0 expected for April.

Thanks to my CRANberries there is also a diff to the previous version. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

24 November 2021

Dirk Eddelbuettel: nanotime 0.3.4 on CRAN: Maintenance Update

Another (minor) nanotime release, now at version 0.3.4, arrived at CRAN overnight. It exports some nanoperiod functionality via a C++ header, and Leonardo and I will use this in an upcoming package that we hope to talk about a little more in a few days. It also adds a few as.character.*() methods that had not been included before. nanotime relies on the RcppCCTZ package for (efficient) high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from a rigorous refactoring by Leonardo who not only rejigged nanotime internals in S4 but also added new S4 types for periods, intervals and durations. The NEWS snippet adds more details.

Changes in version 0.3.4 (2021-11-24)
  • Added a few more as.character conversion function (Dirk)
  • Expose nanoperiod functionality via header file for use by other packages (Leonardo in #95 fixing #94).

Thanks to CRANberries there is also a diff to the previous version. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

25 August 2021

Bits from Debian: DebConf21 welcomes its sponsors!

DebConf21 logo DebConf21 is taking place online, from 24 August to 28 August 2021. It is the 22nd Debian conference, and organizers and participants are working hard together at creating interesting and fruitful events. We would like to warmly welcome the 19 sponsors of DebConf21, and introduce them to you. We have five Platinum sponsors. Our first Platinum sponsor is Lenovo. As a global technology leader manufacturing a wide portfolio of connected products, including smartphones, tablets, PCs and workstations as well as AR/VR devices, smart home/office and data center solutions, Lenovo understands how critical open systems and platforms are to a connected world. Our next Platinum sponsor is Infomaniak. Infomaniak is Switzerland's largest web-hosting company, also offering backup and storage services, solutions for event organizers, live-streaming and video on demand services. It wholly owns its datacenters and all elements critical to the functioning of the services and products provided by the company (both software and hardware). Roche is our third Platinum sponsor. Roche is a major international pharmaceutical provider and research company dedicated to personalized healthcare. More than 100,000 employees worldwide work towards solving some of the greatest challenges for humanity using science and technology. Roche is strongly involved in publicly funded collaborative research projects with other industrial and academic partners and has supported DebConf since 2017. Amazon Web Services (AWS) is our fourth Platinum sponsor. Amazon Web Services is one of the world's most comprehensive and broadly adopted cloud platform, offering over 175 fully featured services from data centers globally (in 77 Availability Zones within 24 geographic regions). AWS customers include the fastest-growing startups, largest enterprises and leading government agencies. Google is our fifth Platinum sponsor. Google is one of the largest technology companies in the world, providing a wide range of Internet-related services and products such as online advertising technologies, search, cloud computing, software, and hardware. Google has been supporting Debian by sponsoring DebConf for more than ten years, and is also a Debian partner sponsoring parts of Salsa's continuous integration infrastructure within Google Cloud Platform. Our Gold sponsor is the Matanel Foundation. The Matanel Foundation operates in Israel, as its first concern is to preserve the cohesion of a society and a nation plagued by divisions. The Matanel Foundation also works in Europe, in Africa and in South America. Our Silver sponsors are: arm: the World s Best SoC Design Portfolio, Arm powered solutions have been supporting innovation for more than 30 years and are deployed in over 160 billion chips to date, Hudson-Trading, a company researching and developing automated trading algorithms using advanced mathematical techniques, Ubuntu the Operating System delivered by Canonical, Globo, the largest media conglomerate in Brazil, founded in Rio de Janeiro in 1925 and distributing high-quality content across multiple platforms, Two Sigma, rigorous inquiry, data analysis, and invention to help solve the toughest challenges across financial services and GitLab, an open source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more. Bronze sponsors: Univention, Gandi.net, daskeyboard, InterFace AG and credativ. And finally, our Supporter level sponsor, ISG.EE. Thanks to all our sponsors for their support! Their contributions make it possible for a large number of Debian contributors from all over the globe to work together, help and learn from each other in DebConf21. Participating in DebConf21 online The 22nd Debian Conference is being held online, due to COVID-19, from August 24 to 28, 2021. Talks, discussions, panels and other activities run from 12:00 to 02:00 UTC. Visit the DebConf21 website at https://debconf21.debconf.org to learn about the complete schedule, watch the live streaming and join the different communication channels for participating in the conference.

20 June 2021

Russ Allbery: Review: The Magician's Nephew

Review: The Magician's Nephew, by C.S. Lewis
Illustrator: Pauline Baynes
Series: Chronicles of Narnia #6
Publisher: Collier Books
Copyright: 1955
Printing: 1978
ISBN: 0-02-044230-0
Format: Mass market
Pages: 186
The Magician's Nephew is the sixth book of the Chronicles of Narnia in the original publication order, but it's a prequel, set fifty years before The Lion, the Witch and the Wardrobe. It's therefore put first in the new reading order. I have always loved world-building and continuities and, as a comics book reader (Marvel primarily), developed a deep enjoyment of filling in the pieces and reconstructing histories from later stories. It's no wonder that I love reading The Magician's Nephew after The Lion, the Witch and the Wardrobe. The experience of fleshing out backstory with detail and specifics makes me happy. If that's also you, I recommend the order in which I'm reading these books. Reading this one first is defensible, though. One of the strongest arguments for doing so is that it's a much stronger, tighter, and better-told story than The Lion, the Witch and the Wardrobe, and therefore might start the series off on a better foot for you. It stands alone well; you don't need to know any of the later events to enjoy this, although you will miss the significance of a few things like the lamp post and you don't get the full introduction to Aslan. The Magician's Nephew is the story of Polly Plummer, her new neighbor Digory Kirke, and his Uncle Andrew, who fancies himself a magician. At the start of the book, Digory's mother is bed-ridden and dying and Digory is miserable, which is the impetus for a friendship with Polly. The two decide to explore the crawl space of the row houses in which they live, seeing if they can get into the empty house past Digory's. They don't calculate the distances correctly and end up in Uncle Andrew's workroom, where Digory was forbidden to go. Uncle Andrew sees this as a golden opportunity to use them for an experiment in travel to other worlds. MAJOR SPOILERS BELOW. The Magician's Nephew, like the best of the Narnia books, does not drag its feet getting started. It takes a mere 30 pages to introduce all of the characters, establish a friendship, introduce us to a villain, and get both of the kids into another world. When Lewis is at his best, he has an economy of storytelling and a grasp of pacing that I wish was more common. It's also stuffed to the brim with ideas, one of the best of which is the Wood Between the Worlds. Uncle Andrew has crafted pairs of magic rings, yellow and green, and tricks Polly into touching one of the yellow ones, causing her to vanish from our world. He then uses her plight to coerce Digory into going after her, carrying two green rings that he thinks will bring people back into our world, and not incidentally also observing that world and returning to tell Uncle Andrew what it's like. But the world is more complicated than he thinks it is, and the place where the children find themselves is an eerie and incredibly peaceful wood, full of grass and trees but apparently no other living thing, and sprinkled with pools of water. This was my first encounter with the idea of a world that connects other worlds, and it remains the most memorable one for me. I love everything about the Wood: the simplicity of it, the calm that seems in part to be a defense against intrusion, the hidden danger that one might lose one's way and confuse the ponds for each other, and even the way that it tends to make one lose track of why one is there or what one is trying to accomplish. That quiet forest filled with pools is still an image I use for infinite creativity and potential. It's quiet and nonthreatening, but not entirely inviting either; it's magnificently neutral, letting each person bring what they wish to it. One of the minor plot points of this book is that Uncle Andrew is wrong about the rings because he's wrong about the worlds. There aren't just two worlds; there are an infinite number, with the Wood as a nexus, and our reality is neither the center nor one of an important pair. The rings are directional, but relative to the Wood, not our world. The kids, who are forced to experiment and who have open minds, figure this out quickly, but Uncle Andrew never shifts his perspective. This isn't important to the story, but I've always thought it was a nice touch of world-building. Where this story is heading, of course, is the creation of Narnia and the beginning of all of the stories told in the rest of the series. But before that, the kids's first trip out of the Wood is to one of the best worlds of children's fantasy: Charn. If the Wood is my mental image of a world nexus, Charn will forever be my image of a dying world: black sky, swollen red sun, and endless abandoned and crumbling buildings as far as the eye can see, full of tired silences and eerie noises. And, of course, the hall of statues, with one of the most memorable descriptions of history and empire I've ever read (if you ignore the racialized description):
All of the faces they could see were certainly nice. Both the men and women looked kind and wise, and they seemed to come of a handsome race. But after the children had gone a few steps down the room they came to faces that looked a little different. These were very solemn faces. You felt you would have to mind your P's and Q's, if you ever met living people who looked like that. When they had gone a little farther, they found themselves among faces they didn't like: this was about the middle of the room. The faces here looked very strong and proud and happy, but they looked cruel. A little further on, they looked crueller. Further on again, they were still cruel but they no longer looked happy. They were even despairing faces: as if the people they belonged to had done dreadful things and also suffered dreadful things.
The last statue is of a fierce, proud woman that Digory finds strikingly beautiful. (Lewis notes in an aside that Polly always said she never found anything specially beautiful about her. Here, as in The Silver Chair, the girl is the sensible one and things would have gone better if the boy had listened to her, a theme that I find immensely frustrating because Susan was the sensible one in the first two books of the series but then Lewis threw that away.) There is a bell in the middle of this hall, and the pillar that holds that bell has an inscription on it that I think every kid who grew up on Narnia knows by heart.
Make your choice, adventurous Stranger;
Strike the bell and bide the danger,
Or wonder, till it drives you mad,
What would have followed if you had.
Polly has no intention of striking the bell, but Digory fights her and does it anyway, waking Jadis from where she sat as the final statue in the hall and setting off one of the greatest reimaginings of a villain in children's literature. Jadis will, of course, become the White Witch who holds Narnia in endless winter some thousand Narnian years later. But the White Witch was a mediocre villain at best, the sort of obvious and cruel villain common in short fairy tales where the author isn't interested in doing much characterization. She exists to be evil, do bad things, and be defeated. She has a few good moments in conflict with Aslan, but that's about it. Jadis in this book is another matter entirely: proud, brilliant, dangerous, and creative. The death of everything on Charn was Jadis's doing: an intentional spell, used to claim a victory of sorts from the jaws of defeat by her sister in a civil war. (I find it fascinating that Lewis puts aside his normally sexist roles here.) Despite the best attempts of the kids to lose her both in Charn and in the Wood (which is inimical to her, in another nice bit of world-building), she manages to get back to England with them. The result is a remarkably good bit of villain characterization. Jadis is totally out of her element, used to a world-spanning empire run with magic and (from what hints we get) vaguely medieval technology. Her plan to take over their local country and eventually the world should be absurd and is played somewhat for laughs. Her magic, which is her great weapon, doesn't even work in England. But Jadis learns at a speed that the reader can watch. She's observant, she pays attention to things that don't fit her expectations, she changes plans, and she moves with predatory speed. Within a few hours in London she's stolen jewels and a horse and carriage, and the local police seem entirely overmatched. There's no way that one person without magic should be a real danger to England around the turn of the 20th century, but by the time the kids manage to pull her back into the Wood, you're not entirely sure England would have been safe. A chaotic confrontation, plus the ability of the rings to work their magic through transitive human contact, ends up with the kids, Uncle Andrew, Jadis, a taxicab driver and his horse all transported through the Wood to a new world. In this case, literally a new world: Narnia at the point of its creation. Here again, Lewis translates Christian myth, in this case the Genesis creation story, into a more vivid and in many ways more beautiful story than the original. Aslan singing the world into existence is an incredible image, as is the newly-created world so bursting with life that even things that normally could not grow will do so. (Which, of course, is why there is a lamp post burning in the middle of the western forest of Narnia for the Pevensie kids to find later.) I think my favorite part is the creation of the stars, but the whole sequence is great. There's also an insightful bit of human psychology. Uncle Andrew can't believe that a lion is singing, so he convinces himself that Aslan is not singing, and thus prevents himself from making any sense of the talking animals later.
Now the trouble about trying to make yourself stupider than you really are is that you very often succeed.
As with a lot in Lewis, he probably meant this as a statement about faith, but it generalizes well beyond the religious context. What disappointed me about the creation story, though, is the animals. I didn't notice this as a kid, but this re-read has sensitized me to how Lewis consistently treats the talking animals as less than humans even though he celebrates them. That happens here too: the newly-created, newly-awakened animals are curious and excited but kind of dim. Some of this is an attempt to show that they're young and are just starting to learn, but it also seems to be an excuse for Aslan to set up a human king and queen over them instead of teaching them directly how to deal with the threat of Jadis who the children inadvertently introduced into the world. The other thing I dislike about The Magician's Nephew is that the climax is unnecessarily cruel. Once Digory realizes the properties of the newly-created world, he hopes to find a way to use that to heal his mother. Aslan points out that he is responsible for Jadis entering the world and instead sends him on a mission to obtain a fruit that, when planted, will ward Narnia against her for many years. The same fruit would heal his mother, and he has to choose Narnia over her. (It's a fairly explicit parallel to the Garden of Eden, except in this case Digory passes.) Aslan, in the end, gives Digory the fruit of the tree that grows, which is still sufficient to heal his mother, but this sequence made me angry when re-reading it. Aslan knew all along that what Digory is doing will let him heal his mother as well, but hides this from him to make it more of a test. It's cruel and mean; Aslan could have promised to heal Digory's mother and then seen if he would help Narnia without getting anything in return other than atoning for his error, but I suppose that was too transactional for Lewis's theology or something. Meh. But, despite that, the only reason why this is not the best Narnia book is because The Voyage of the Dawn Treader is the only Narnia book that also nails the ending. The Magician's Nephew, up through Charn, Jadis's rampage through London, and the initial creation of Narnia, is fully as good, perhaps better. It sags a bit at the end, partly because it tries to hard to make the Narnian animals humorous and partly because of the unnecessary emotional torture of Digory. But this still holds up as the second-best Narnia book, and one I thoroughly enjoyed re-reading. If anything, Jadis and Charn are even better than I remembered. Followed by the last book of the series, the somewhat notorious The Last Battle. Rating: 9 out of 10

10 June 2021

Vincent Bernat: Serving WebP & AVIF images with Nginx

WebP and AVIF are two image formats for the web. They aim to produce smaller files than JPEG and PNG. They both support lossy and lossless compression, as well as alpha transparency. WebP was developed by Google and is a derivative of the VP8 video format.1 It is supported on most browsers. AVIF is using the newer AV1 video format to achieve better results. It is supported by Chromium-based browsers and has experimental support for Firefox.2

Your browser supports WebP and AVIF image formats. Your browser supports none of these image formats. Your browser only supports the WebP image format. Your browser only supports the AVIF image format.

Without JavaScript, I can t tell what your browser supports.

Converting and optimizing images For this blog, I am using the following shell snippets to convert and optimize JPEG and PNG images. Skip to the next section if you are only interested in the Nginx setup.

JPEG images JPEG images are converted to WebP using cwebp.
find media/images -type f -name '*.jpg' -print0 \
    xargs -0n1 -P$(nproc) -i \
      cwebp -q 84 -af ' ' -o ' '.webp
They are converted to AVIF using avifenc from libavif:
find media/images -type f -name '*.jpg' -print0 \
    xargs -0n1 -P$(nproc) -i \
      avifenc --codec aom --yuv 420 --min 20 --max 25 ' ' ' '.avif
Then, they are optimized using jpegoptim built with Mozilla s improved JPEG encoder, via Nix. This is one reason I love Nix.
jpegoptim=$(nix-build --no-out-link \
      -E 'with (import <nixpkgs> ); jpegoptim.override   libjpeg = mozjpeg;  ')
find media/images -type f -name '*.jpg' -print0 \
    sort -z
    xargs -0n10 -P$(nproc) \
      $ jpegoptim /bin/jpegoptim --max=84 --all-progressive --strip-all

PNG images PNG images are down-sampled to 8-bit RGBA-palette using pngquant. The conversion reduces file sizes significantly while being mostly invisible.
find media/images -type f -name '*.png' -print0 \
    sort -z
    xargs -0n10 -P$(nproc) \
      pngquant --skip-if-larger --strip \
               --quiet --ext .png --force
Then, they are converted to WebP with cwebp in lossless mode:
find media/images -type f -name '*.png' -print0 \
    xargs -0n1 -P$(nproc) -i \
      cwebp -z 8 ' ' -o ' '.webp
No conversion is done to AVIF: lossless compression is not as efficient as pngquant and lossy compression is only marginally better than what I get with WebP.

Keeping only the smallest files I am only keeping WebP and AVIF images if they are at least 10% smaller than the original format: decoding is usually faster for JPEG and PNG; and JPEG images can be decoded progressively.3
for f in media/images/**/*. webp,avif ; do
  orig=$(stat --format %s $ f%.* )
  new=$(stat --format %s $f)
  (( orig*0.90 > new ))   rm $f
done
I only keep AVIF images if they are smaller than WebP.
for f in media/images/**/*.avif; do
  [[ -f $ f%.* .webp ]]   continue
  orig=$(stat --format %s $ f%.* .webp)
  new=$(stat --format %s $f)
  (( $orig > $new ))   rm $f
done
We can compare how many images are kept when converted to WebP or AVIF:
printf "     %10s %10s %10s\n" Original WebP AVIF
for format in png jpg; do
  printf " $ format:u  %10s %10s %10s\n" \
    $(find media/images -name "*.$format"   wc -l) \
    $(find media/images -name "*.$format.webp"   wc -l) \
    $(find media/images -name "*.$format.avif"   wc -l)
done
AVIF is better than MozJPEG for most JPEG files while WebP beats MozJPEG only for one file out of two:
       Original       WebP       AVIF
 PNG         64         47          0
 JPG         83         40         74

Further reading I didn t detail my choices for quality parameters and there is not much science in it. Here are two resources providing more insight on AVIF:

Serving WebP & AVIF with Nginx To serve WebP and AVIF images, there are two possibilities:
  1. use <picture> to let the browser pick the format it supports, or
  2. use content negotiation to let the server send the best-supported format.
I use the second approach. It relies on inspecting the Accept HTTP header in the request. For Chrome, it looks like this:
Accept: image/avif,image/webp,image/apng,image/*,*/*;q=0.8
I configure Nginx to serve AVIF image, then the WebP image, and fallback to the original JPEG/PNG image depending on what the browser advertises:4
http  
  map $http_accept $webp_suffix  
    default        "";
    "~image/webp"  ".webp";
   
  map $http_accept $avif_suffix  
    default        "";
    "~image/avif"  ".avif";
   
 
server  
  # [ ]
  location ~ ^/images/.*\.(png jpe?g)$  
    add_header Vary Accept;
    try_files $uri$avif_suffix$webp_suffix $uri$avif_suffix $uri$webp_suffix $uri =404;
   
 
For example, let s suppose the browser requests /images/ont-box-orange@2x.jpg. If it supports WebP but not AVIF, $webp_suffix is set to .webp while $avif_suffix is set to the empty string. The server tries to serve the first existing file in this list:
  • /images/ont-box-orange@2x.jpg.webp
  • /images/ont-box-orange@2x.jpg
  • /images/ont-box-orange@2x.jpg.webp
  • /images/ont-box-orange@2x.jpg
If the browser supports both AVIF and WebP, Nginx walks the following list:
  • /images/ont-box-orange@2x.jpg.webp.avif (it never exists)
  • /images/ont-box-orange@2x.jpg.avif
  • /images/ont-box-orange@2x.jpg.webp
  • /images/ont-box-orange@2x.jpg
Eugene Lazutkin explains in more detail how this works. I have only presented a variation of his setup supporting both WebP and AVIF.

  1. VP8 is only used for lossy compression. Lossless compression is using an unrelated format.
  2. Firefox support was scheduled for Firefox 86 but because of the lack of proper color space support, it is still not enabled by default.
  3. Progressive decoding is not planned for WebP but could be implemented using low-quality thumbnail images for AVIF. See this issue for a discussion.
  4. The Vary header ensures an intermediary cache (a proxy or a CDN) checks the Accept header before using a cached response. Internet Explorer has trouble with this header and may not be able to cache the resource properly. There is a workaround but Internet Explorer s market share is now so small that it is pointless to implement it.

Dirk Eddelbuettel: #33: Collaborative Editing and Execution in Shared Byoby Sessions

Welcome to the 33th post in the rigorously raconteuring R recommendations series, or R4 for short. This post is also a post in the T4 series of tips, tricks, tools, and toys as it picks up and extends earlier posts on byobu. And it fits nicely in the more recent ESS-Intro series as we show some Emacs. You can find earlier R4 posts here, and the T4 posts here; the ESS-Intro series is here. The focus of this short video (and slides) is on collaboration using files, but also entire sessions, execution and all aspects of joint exploration, development or debugging. Anything you can do in a terminal you can also do shared in a terminal. The video contains a brief lightning talk, and a shared session jointly with Grant McDermott and Vicent Arel-Bundock. My big big thanks to both of them for prodding and encouragement, as well as fearless participation in the joint section of the video:

The corresponding pdf slides are here. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

20 March 2021

Dirk Eddelbuettel: An Ode to Stable Interfaces: R and R Core Deserve So Much Praise

A few days ago, a friend and I were riffing about the wonderful stability of R and (subsets of) R packages. The rigorous ASAN/UBSAN/Valgrind/ checks, while at times frustrating for us package maintainers when we do not have easily replicable setups [1], really help in ensuring code quality. As do of course all other layers of quality control at CRAN, and for R. In passing, I mentioned there was an older blog post demonstrating a little power-law-alike behaviour between the most frequent R Core committer and everybody else. So I was intrigued. Could we just pick up a blog post I had written in August of 2007, or almost fourteen years ago, and run it as is? [2] Yes, we can. Which is truly, truly awesome. Back then I must have taken a minor shortcut and analysed just one calendar year of SVN that was pre-extracted (and a few more still exists here if one scrolls down). Maybe then I might not have had the r-devel SVN repo checkout. But these days (and for probably a decade now) I do, and just a few lines of bash get us a full log:
#!/bin/bash

## adjust as needed
svn=$ HOME /svn/r-devel

rev=$(cd $ svn  && svn info --show-item revision)
today=$(date +%Y-%m-%d)

echo -n "Extracting $ rev  revisions at $ today  ... "
(cd $ svn  && svn log --limit $ rev  )   gzip -9 > svn-log-$ today .txt.gz
echo "done"
So that leads to one code adjustments given the different input source. But otherwise the first paragraph runs as is (and now gives us 49.2% for the amazing Prof Ripley):
logfile <- "svn-log-2021-03-20.txt.gz"

## cf http://dirk.eddelbuettel.com/blog/2007/08/11/
x <- readLines(logfile)
rx <- x[grep("^r",x)]
who <- gsub(" ","",sapply(strsplit(rx,"\\ "),"[",2))
twho <- table(who)
twho["ripley"]/sum(twho)
That is what one gets by trusting stable interfaces: code untouched for fourteen years runs unchanged. R itself has had well over sixty releases since then, including two major and eighteen minor releases. Yet the code just runs, including the code for two graphs one can reproduce with the exact same code as we show next.
tod <- unlist(sapply(rx,function(x)strsplit(x,split=" ")[[1]][6]))
tod <- tod[who=="ripley"]

tz <- sub(pattern=".*(-[0-9] 4 ).*",replacement="\\1",x=rx)
tz <- tz[who=="ripley"]
tz <- as.numeric(tz)/100
offset <- 3600*tz

z <- strptime(tod,format="%H:%M:%S")
hist(z,"hours",main="Ripley Commit Times in SVN TZ")

h <- z - offset
h <- format(h,format="%H")
h <- factor(as.numeric(h), levels=0:23)
## added as.vector() here to suppress a warning
dotchart(as.vector(table(h)), main="Ripley Commit Times, By Hour in GMT",
         labels=paste(0:23,1:24,sep=":"))
The code reproduces the chart from 2008, but this time uses the full twenty plus years of SVN history. I added just one as.vector() to suppress one new warning which appears under current R and which was presumably added in the fourteen years since (at the chart is produces without it too). The remainder of the code also runs. I just added one library(zoo) my blog post had omitted. No other changes.
## rather extract both  date and time
dat <- unlist(sapply(rx, function(x)  
  txt <- strsplit(x,split=" ")[[1]]
  paste(txt[5], txt[6])
 ))
## subset on Prof Ripley
dat <- dat[who == "ripley"]
## and convert to POSIXct, correcting by tz as well
datpt <- as.POSIXct(strptime(dat,format="%Y-%m-%d %H:%M:%S")) - offset

## turn into zoo -- we use a constant series of ones as each
## committ is taken as a timestamped event
library(zoo)
datzoo <- zoo(1, order.by=datpt)
## and use zoo to aggregate into commits per date
daily <- aggregate(datzoo, as.Date(index(datzoo)), sum)

## now plot as grey bars
plot(daily, col='darkgrey', type='h', lwd=2,
     ylab="Nb of SVN commits, three-week median",
     xlab="R release dates 2.5.0 and 2.5.1 shown in orange",
     main="The amazing Prof. Ripley")
## mark the two R releases of 2007
abline(v=c(as.Date("2007-04-24"),as.Date("2007-06-28")),col='orange',lwd=1.5)
## and do a quick centered rolling median
lines(rollmedian(daily, 21, align="center"), lwd=3)
It produces this chart spanning two decades of commits. [3] The subtitle highlighting the then-most-recent releases is a little quaint now given that R has had eighten major.minor releases, and over sixty total releases, since then. Stable and rigourously maintained interfaces are a fantastic resource that is dramatically under-appreciated. Efforts such as the ten-year reproduction challenge demonstrate that this really is not a given. Maybe instead of celebrating band aides ( look, I reproduce via code I have frozen in a virtual environment / container / machine / ) we should celebrate languages, ecosystems, packages, that allow us to rely on just the code itself. Because we can. And we should strengthen and reinforce that ability. And discourage rapid changes just for changes sake. Code running for a decade, or even longer, is a huge boon to everybody relying on it. Three cheers to R Core. [1] Docker containers would be really good, and a step above the specs in the README. Winston s nice r-debug sumo container comes closest and helps a lot, and is updated regularly (which my earlier r-devel-san container is not). [2] The post owes some of its code ideas to Ben Bolker and Simon Jackman, but links to now-stale prior affiliations of theirs. [3] And the singularly impressive contributions charted remain unparalled, but were already the focus of the previous post. Yet over three times as a long period, they remain even more stunning. Edit 2021-03-21: Two minor fixes for grammar and typing.

27 January 2021

Russ Allbery: Review: A Deadly Education

Review: A Deadly Education, by Naomi Novik
Series: The Scholomance #1
Publisher: Del Rey
Copyright: 2020
ISBN: 0-593-12849-4
Format: Kindle
Pages: 319
Some children are born with magic, which grows as they mature. Magic attracts maleficaria: extremely deadly magical beasts that want to feast on that magic. Having innate magical ability is therefore a recipe for endless attacks from monsters and a death at a young age. This was true even for the enclaves, which are the rich, gated communities of the magical world. Hence, the Scholomance. This is a boarding school for magic users placed in the Void and protected against maleficaria as completely as possible while still letting the students graduate and leave after their senior year. Students are sent there via a teleportation spell with a weight allowance, taught magic by automated systems and magical artifacts, and left on their own to make alliances and survive. Or not survive; protected as well as possible still means that there are maleficaria everywhere, sneaking past the wards of the graduation hall and looking for snacks. The school sends cleansing fire through the halls at certain times; the rest of the time, the students either learn enough magic to defeat maleficaria themselves, form alliances with those who can, or die to feed the magic of the school. Enter Galadriel, or El as she prefers. She's not an enclave kid; she's the grumpy, misfit daughter of a hippie mother whose open-hearted devotion to healing and giving away her abilities make her the opposite of the jealously guarded power structures of the enclaves. El has no resources other than what she can muster on her own. She also has her mother's ethics, which means that although she has an innate talent for malia, drawing magic from the death of other living things, she forces herself to build her mana through rigorously ethical means. Like push-ups. Or, worse, crochet. At the start of the book, El is in her third year of four, and significantly more of her classmates are alive than normally would be. That's because of her classmate, Orion Lake, who has made a full-time hobby of saving everyone from maleficaria. His unique magical ability frees him from the constraints of mana or malia that everyone else is subject to, and he uses that to be a hero, surrounded by adoring fans. And El is thoroughly sick of it. This book is so good in so many different ways that I don't know where to start. Obviously, A Deadly Education is a twist on the boarding school novel, both the traditional and the magical kind. This is not a genre in which I'm that well-read, but even with my lack of familiarity, I noticed so many things Novik does to improve the genre tropes, starting with not making the heroic character with the special powers the protagonist. And getting rid of all the adults, which leaves way more space for rich social dynamics between the kids (complex and interesting ones that are entangled with the social dynamics outside of the school, not some simplistic Lord of the Flies take). Going alone anywhere in the school is dangerous, as is sitting at the bad tables in the cafeteria, so social cliques become a matter of literal life and death. And the students aren't just trying to survive; the ones who aren't part of enclaves are jockeying for invitations or trying to build the power to help their family and allies form their own. El is the first-person narrator of the story and she's wonderful. She's grumpy, cynical, and sarcastic, which is often good for first-person narrators, but she also has a core of ethics from her mother, and from her own decisions, that gives her so much depth. She is the type of person who knows exactly how much an ethical choice will cost her and how objectively stupid it is, and then will make it anyway out of sheer stubbornness and refuse to take credit for it. I will happily read books about characters like El until the end of time. Her mother never appears in this book, and yet she's such a strong presence because El's relationship with her matters, to both El and to the book. El could not be more unlike her mother in both personality and in magical focus, and she's exasperated by the sheer impracticality of some of her mother's ideals. And yet there's a core of love and understanding beneath that, a level at which El completely understands her mother's goals, and El relies on it even when she doesn't realize she's doing so. I don't think I've ever read a portrayal of a mother-daughter relationship this good where one of the parties isn't even present. And I haven't even gotten to the world-building, and the level to which Novik chases down and explores all the implications of this ridiculous murder machine of a school. I will offer this caveat: If you poke at the justification for creating this school in the way it was built, it's going to teeter a lot. That society thought this school was the best solution to its child mortality problem is just something you have to roll with. But once you accept that, the implications are handled so very well. The school is an inhuman character in its own right, with exasperating rules that the students learn and warn each other about. It tries to distract you with rare spellbooks or artifact materials because it's trying to kill you. The language tapes whisper horrific stories of your death. The back wall of your room is a window to the Void, from which you can demand spellbooks. You'll even get them in languages that you understand, for a generous definition of understand that may have involved glancing at one page of text, so be careful not to do that! The school replaces all of the adult teachers in the typical boarding school novel and is so much more interesting than any of them because it adds the science fiction thrill of setting as character. The world-building does mean a lot of infodumping, so be prepared for that. El likes to explain things, tell stories, and over-analyze her life, and reading this book is a bit like reading the journal of a teenage girl. For me, El's voice is so strong, authentic, stubborn, and sarcastically funny that I scarcely noticed the digressions into background material. And the relationships! Some of the turns will be predictable, since of course El's stubborn ethics will be (eventually) rewarded by the story, but the dynamic that develops between El and Orion is something special. It takes a lot to make me have sympathy with the chosen one boy hero, but Novik pulls it off without ever losing sight of the dynamics of class and privilege that are also in play. And the friendships El develops almost accidentally by being stubbornly herself are just wonderful, and the way she navigates them made me respect her even more. The one negative thing I will say about this book is that I don't think Novik quite nailed the climax. Some of this is probably because this is the first book of a series and Novik wanted to hold some social developments in reserve, but I thought El got a bit sidelined and ended up along for the ride in an action-movie sequence. Still, it's a minor quibble, and it's clear from the very end of the book that El is going to get more attention and end up in a different social position in the next book. This was a wholly engrossing and enjoyable story with a satisfying climax and only the barb of a cliffhanger in the very last line. It's the best SFF novel published in 2020 that I've read so far (yes, even better than Network Effect). Highly recommended, and I hope it gets award recognition this year. Followed by The Last Graduate (not yet published at the time of this review). Rating: 9 out of 10

26 January 2021

Russ Allbery: Review: The City We Became

Review: The City We Became, by N.K. Jemisin
Series: The Great Cities Trilogy #1
Publisher: Orbit
Copyright: March 2020
ISBN: 0-316-50985-X
Format: Kindle
Pages: 449
At an unpredictable point after a city has accumulated enough history, character, and sense of itself, it is born as a living creature. It selects an avatar who both embodies the city and helps it be born. But in that moment of birth, a city is at its most vulnerable, and that is when the Enemy attacks. The birth of cities and the Enemy attacks have happened for millennia, so the cities that have survived the process have developed a system. The most recently born goes to assist the avatar with the birthing process and help fight off Enemy attacks. But the process has become riskier and the last two cities have failed to be born, snuffed out in natural disasters despite that support. Now, it's New York City's turn. It selects its avatar and survives the initial assault with the help of S o Paulo. But something still goes wrong, and it is severely injured in the process. Complicating matters, now there are five more avatars, one for each borough, who will have to unite to fight off the Enemy. And, for the first time, the Enemy has taken human form and is attacking with reason and manipulation and intelligence, not just force. The City We Became has a great premise: take the unique sense of place that defines a city and turn it into a literalized character in a fantasy novel. The avatars are people who retain their own lives and understanding (with one exception that I'll get to in a moment), but gain an awareness of the city they represent. They can fight and repair their city through sympathetic magic and metaphor made real. The prelude that introduces this concept (adapted from Jemisin's earlier short story "The City Born Great") got too gonzo for me, but once Jemisin settles into the main story and introduces avatars with a bit more distance from the city they represent, the premise clicked. The execution, on the other hand, I thought was strained. The biggest problem is that the premise requires an ensemble cast of five borough avatars, the primary avatar, S o Paulo, and the Enemy. That's already a lot, but for the story to work each avatar has to be firmly grounded in their own unique experience of New York, which adds family members, colleagues, and roommates. That's too much to stuff into one novel, which means characters get short shrift. For example, Padmini, the avatar of Queens, gets a great introductory scene and a beautiful bit of characterization that made her one of my favorite characters, but then all but disappears for the remainder of the book. She's in the scenes, but not in a way that matters. Brooklyn and Aislyn get moments of deep characterization, but there's so much else going on that they felt rushed. And what ever happened to Manny's roommate? The bulk of the characterization in this book goes to Broncha, the Bronx avatar, a Lenape woman and a tough-as-nails administrator of a community art museum and maker space. The dynamics between her and her co-workers, her mentorship of Veneza, and her early encounters with the Woman in White are my favorite parts of the book. I thought she and Brooklyn were a useful contrast: two very different ways of finding a base of power in the city without letting go of one's ideals. But before we get to Broncha, we first meet Manny, the Manhattan avatar. Thematically, I thought what Jemisin did here was extremely clever. Manny's past is essentially erased at the start of the book, so he's making sense of this world in the same way the reader is. It makes him a useful reader insert character and parallels the typical tourist experience of arriving in Manhattan and trying to use it to make sense of New York. He's disconnected from the rest of the city because he's the dangerous newcomer with power but not a lot of understanding, which works with my model of the political dynamics of Manhattan. Unfortunately, he's not an interesting person. I appreciated what was happening at the metaphorical layer, but Manny veers between action hero and exposition prompt, and his amnesia meant I never got enough sense of him as a character to care that much about what happened to him. I thought his confrontation with the Woman in White near the start of the book, which establishes the major villain of the book, felt clunky and forced compared to her later encounters with the other characters. The Woman in White, though, is a great villain. It's clear from earlier on that the Enemy is Lovecraftian, but the Woman in White mixes mad scientist glee, manic enthusiasm, and a child-like amusement at the weirdness of humanity into the more typical tropes of tentacles, corruption, and horrific creatures. One of my qualms about reading this book is that I'm not a horror fan and don't enjoy the mental images of unspeakable monsters, but the Woman in White puts such a fascinating spin on them that I enjoyed the scenes in which she appeared. I think the book was at its best when she was trying to psychologically manipulate the characters or attack them with corrupted but pre-existing power structures. I was less interested when it turned into an action-movie fight against animated monsters. The other place Jemisin caught me by surprise is too much of a spoiler to describe in detail (and skip the next paragraph in its entirety if you want to avoid all spoilers): Jemisin didn't take the moral conflict of the book in the direction I was expecting. This book is more interested in supporting the people who are already acting ethically than in redeeming people who make bad choices. That produces a deus ex machina ending that's a bit abrupt, but I appreciated the ethical stance. Overall, I thought the premise was great but the execution was unsteady and a bit overstuffed. There are some great characters and some great scenes, but to me they felt disjointed and occasionally rushed. You also need to enjoy characters taking deep pride in the feel of a specific place and advocating for it with the vigor of a sports rivalry, along with loving descriptions of metaphors turned into magical waves of force. But, if you can roll with that, there are moments of real awe. Jemisin captured for me the joy that comes from a deeply grounded sense of connection to a place. Recommended, albeit with caveats, if you're in the mood for reading about people who love the city they live in. This is the first book of a planned trilogy and doesn't resolve the main conflict, but it reaches a satisfying conclusion. The title of the next book has not yet been announced at the time of this review. Rating: 7 out of 10

10 December 2020

John Goerzen: How the Attention Economy Hurts You via Social Media Sites like Facebook

There is a whole science to manipulating our attention. And because there is a lot of money to be made by doing this well, it means we all encounter attempts to manipulate what we pay attention to each day. What is this, and how is it harmful? This post will be the first on a series on the topic. Why is attention so important? When people use Facebook, they use it for free. Facebook generally doesn t even try to sell them anything, yet has billions in revenues. What, then, is Facebook s product? Well, really, it s you. Or, more specifically, your attention. Facebook sells your attention to advertisers. Everything they do is in service to that. They want you to spend more time on the site so they can show you more ads. (I should say here that I m using Facebook as an example, but this applies to other social media companies too.) Seeking to maximize attention So if your attention is so important to their profit, it follows naturally that they would seek ways to get people to spend more time on their site. And they do. They track all sorts of metrics, including engagement (if you click like , comment, share, or otherwise interact with content). They know which sorts of things are likely to capture your (and I mean you in specific!) attention and show you that. Your neighbor may have different interests and Facebook judges different things are likely to capture their attention. Manipulating your attention Attention turning into money isn t unique for social media. In fact, in the article If It Bleeds, It Leads: Understanding Fear-Based Media, Psychology Today writes:
In previous decades, the journalistic mission was to report the news as it actually happened, with fairness, balance, and integrity. However, capitalistic motives associated with journalism have forced much of today s television news to look to the spectacular, the stirring, and the controversial as news stories. It s no longer a race to break the story first or get the facts right. Instead, it s to acquire good ratings in order to get advertisers, so that profits soar. News programming uses a hierarchy of if it bleeds, it leads. Fear-based news programming has two aims. The first is to grab the viewer s attention. In the news media, this is called the teaser. The second aim is to persuade the viewer that the solution for reducing the identified fear will be in the news story. If a teaser asks, What s in your tap water that YOU need to know about? a viewer will likely tune in to get the up-to-date information to ensure safety.
You ve probably seen fear-based messages a lot on Facebook. They will highlight messages to liberals about being afraid of what Trump is doing, and to conservatives about being afraid of what Biden is doing. They may or may not even intentionally be doing this; it is their algorithm predicts that those would maximize time and engagement for certain people, so that s what they see. Fear leads to controversy It s not just fear, though. Social media also loves controversy. There s nothing that makes people really want to stay on Facebook like anger. See something controversial and you ll see hundreds or thousands of people are there arguing about it and in the process, giving Facebook their attention. A quick Internet search will show you numerous articles on how marketing companes can leverage controvery to get attention and engagement with their campaigns. Consequences of maximizing fear and controversy What does it mean to society at large and to you personally that large companies make a lot of money by maximizing fear and controversy? The most obvious way is it leads to less common ground. If the posts and reactions that show common ground are never seen because they don t drive engagement, it poisons the well; left and right hate each other with ever more vigor a profitable outcome to Facebook, but a poisonous one to all of us. I have had several friendships lost because I a liberal in agreement with these friends on political matters still talk to Trump voters. On the other side, we ve seen people storm the Michigan statehouse with weapons. How did that level of disagreement and even fear behind it get so firmly embedded in our society? Surely the fact that social media shows us things designed to stimulate fear and anger must play a role. What does it do to our ability to have empathy for, and understand, others? The Facebook groups I ve been in for like-minded people have largely been flooded with memes calling the President rump and other things clearly designed to make people angry or fearful. It s a worthless experience, and not just that, but it s a harmful experience. When our major media TV and social networks all are optimizing for fear, anger, and controvesry, we have a society beholden to fear, anger, and controvesy. In my next installment, I m going to talk about what to do about this, including the decentralized social networks of the Fediverse that are specifically designed to put you back in charge of your attention. Update 2020-12-16: There are two followup articles for this: how to join the Fediverse and non-creepy technology purchasing and gifting guides. The latter references the FSF s page on software manipulation towards addiction, which is particularly relevant to this topic.

19 September 2020

Russell Coker: Burning Lithium Ion Batteries

I had an old Nexus 4 phone that was expanding and decided to test some of the theories about battery combustion. The first claim that often gets made is that if the plastic seal on the outside of the battery is broken then the battery will catch fire. I tested this by cutting the battery with a craft knife. With every cut the battery sparked a bit and then when I levered up layers of the battery (it seems to be multiple flat layers of copper and black stuff inside the battery) there were more sparks. The battery warmed up, it s plausible that in a confined environment that could get hot enough to set something on fire. But when the battery was resting on a brick in my backyard that wasn t going to happen. The next claim is that a Li-Ion battery fire will be increased with water. The first thing to note is that Li-Ion batteries don t contain Lithium metal (the Lithium high power non-rechargeable batteries do). Lithium metal will seriously go off it exposed to water. But lots of other Lithium compounds will also react vigorously with water (like Lithium oxide for example). After cutting through most of the center of the battery I dripped some water in it. The water boiled vigorously and the corners of the battery (which were furthest away from the area I cut) felt warmer than they did before adding water. It seems that significant amounts of energy are released when water reacts with whatever is inside the Li-Ion battery. The reaction was probably giving off hydrogen gas but didn t appear to generate enough heat to ignite hydrogen (which is when things would really get exciting). Presumably if a battery was cut in the presence of water while in an enclosed space that traps hydrogen then the sparks generated by the battery reacting with air could ignite hydrogen generated from the water and give an exciting result. It seems that a CO2 fire extinguisher would be best for a phone/tablet/laptop fire as that removes oxygen and cools it down. If that isn t available then a significant quantity of water will do the job, water won t stop the reaction (it can prolong it), but it can keep the reaction to below 100C which means it won t burn a hole in the floor and the range of toxic chemicals released will be reduced. The rumour that a phone fire on a plane could do a China syndrome type thing and melt through the Aluminium body of the plane seems utterly bogus. I gave it a good try and was unable to get a battery to burn through it s plastic and metal foil case. A spare battery for a laptop in checked luggage could be a major problem for a plane if it ignited. But a battery in the passenger area seems unlikely to be a big problem if plenty of water is dumped on it to prevent the plastic case from burning and polluting the air. I was not able to get a result that was even worthy of a photograph. I may do further tests with laptop batteries.

13 August 2020

Erich Schubert: Publisher MDPI lies to prospective authors

The publisher MDPI is a spammer and lies. If you upload a paper draft to arXiv, MDPI will send spam to the authors to solicit submission. Within minutes of an upload I received the following email (sent by MDPI staff, not some overly eager new editor):
We read your recent manuscript "[...]" on
arXiv, and sincerely invite you to submit it to our journal Future
Internet, if it has not been published or submitted elsewhere.
Future Internet (ISSN 1999-5903, indexed by Scopus, Ei compendex,
*ESCI*-Web of Science) is a journal on Internet technologies and the
information society. It maintains a rigorous and fast peer review system
with a median publication time of 35 days from submission to online
publication, and 3 days from acceptance to publication. The journal
scope is shown here:
https://www.mdpi.com/journal/futureinternet/about.
Editorial Board: https://www.mdpi.com/journal/futureinternet/editors.
Since Future Internet is an open access journal there is a publication
fee. Your paper will be published, with a 20% discount (amounting to 200
CHF), and provided that it is accepted after our standard peer-review
procedure. 
First of all, the email begins with a lie. Because this paper clearly states that it is submitted elsewhere. Also, it fits other journals much better, and if they had read even just the abstract, they would have known. This is predatory behavior by MDPI. Clearly, it is just about getting as many submissions as possible. The journal charges 1000 CHF (next year, 1400 CHF) to publish the papers. Its about the money. Also, there have been reports that MDPI ignores the reviews, and always publishes even when reviewers recommended rejection The reviewer requests I have received from MDPI came with unreasonable deadlines, which will not allow for a thorough peer review. Hence I asked to not ever be emailed by them again. I must assume that many other qualified reviewers do the same. MDPI boasts in their 2019 annual report a median time to first decision of 19 days in my discipline the typical time window to ask for reviews is at least a month (for shorter conference papers, not full journal articles), because professors tend to have lots of other duties, hence they need more flexibility. Above paper has been submitted in March, and is now under review for 4 months already. This is an annoying long time window, and I would appreciate if this were less, but it shows how extremely short the MDPI time frame is. They also claim 269.1k submissions and 106.2k published papers, so the acceptance rate is around 40% on average, and assuming that there are some journals with higher standards there then some must have acceptance rates much higher than this. I d assume that many reputable journals have 40% desk-rejection rate for papers that are not even on-topic The average cost to authors is given as 1144 CHF (after discounts, 25% waived feeds etc.), so they, so we are talking about 120 million CHF of revenue from authors. Is that what you want academic publishing to be? I am not happy with some of the established publishers such as Elsevier that also overcharge universities heavily. I do think we need to change academic publishing, and arXiv is a big improvement here. But I do not respect publishers such as MDPI that lie and send spam.

13 May 2020

Russ Allbery: Review: Gideon the Ninth

Review: Gideon the Ninth, by Tamsyn Muir
Series: The Locked Tomb #1
Publisher: Tor
Copyright: September 2019
ISBN: 1-250-31317-1
Format: Kindle
Pages: 448
Despite being raised there, Gideon Nav is an outsider in the Ninth House. Her mother, already dead, fell from the sky with a one-day-old Gideon in tow, leaving her an indentured servant. She's a grumpy, caustic teenager in a world of moldering corpses, animated skeletons, and mostly-dead adults whose parts are falling off. Her world is sword fighting, dirty magazines, a feud with the house heir Harrowhark, and a determination to escape the terms of her indenture. Gideon does get off the planet, but not the way that she expects. She doesn't get accepted into the military. She ends up in the middle of a bizarre test, or possibly an ascension rite, mingling with and competing with the nobility of the empire alongside her worst enemy. I struggled to enjoy the beginning of Gideon the Ninth. Gideon tries to carry the story on pure snark, but it is very, very goth. If you like desiccated crypts, mostly-dead goons, betrayal, frustration, necromancers, black robes, disturbing family relationships, gloom, and bitter despair, the first six chapters certainly deliver, but I was sick of it by the time Gideon gets out. Thankfully, the opening is largely unlike the rest of the book. What starts as an over-the-top teenage goth rebellion turns into a cross between a manor house murder mystery and a competitive escape room. This book is a bit of a mess, but it's a glorious mess. It's also the sort of glorious mess that I don't think would have been written or published twenty years ago, and I have a pet theory that attributes this to the invigorating influence of fanfic and writers who grew up reading and writing it. I read a lot of classic science fiction and epic fantasy as a teenager. Those books have many merits, obviously, but emotional range is not one of them. There are a few exceptions, but on average the genre either focused on puzzles and problem solving (how do we fix the starship, how do we use the magic system to take down the dark god) or on the typical "heroic" (and male-coded) emotions of loyalty, bravery, responsibility, authority, and defiance of evil. Characters didn't have messy breakups, frenemies, anxiety, socially-awkward love affairs, impostor syndrome, self-hatred, or depression. And authors weren't allowed to fall in love with the messiness of their characters, at least on the page. I'm not enough of a scholar to make the argument well, but I suspect there's a case to be made that fanfic exists partially to fill this gap. So much of fanfic starts from taking the characters on the canonical page or screen and letting them feel more, live more, love more, screw up more, and otherwise experience a far wider range of human drama, particularly compared to what made it into television, which was even more censored than what made it into print. Some of those readers and writers are now writing for publication, and others have gone into publishing. The result, in my theory, is that the range of stories that are acceptable in the genre has broadened, and the emotional texture of those stories has deepened. Whether or not this theory is correct, there are now more novels like this in the world, novels full of grudges, deflective banter, squabbling, messy emotional processing, and moments of glorious emotional catharsis. This makes me very happy. To describe the emotional payoff of this book in any more detail would be a huge spoiler; suffice it to say that I unabashedly love fragile competence and unexpected emotional support, and adore this book for containing it. Gideon's voice, irreverent banter, stubborn defiance, and impulsive good-heartedness are the center of this book. At the start, it's not clear whether there will be another likable character in the book. There will be, several of them, but it takes a while for Gideon to find them or for them to become likable. You'll need to like Gideon well enough to stick with her for that journey. I read books primarily for the characters, not for the setting, and Gideon the Ninth struck some specific notes that I will happily read endlessly. If that doesn't match your preferences, I would not be too surprised to hear you bounced off the book. There's a lot here that won't be to everyone's taste. The setting felt very close to Warhammer 40K: an undead emperor that everyone worships, endless war, necromancy, and gothic grimdark. The stage for most of the book is at least more light-filled, complex, and interesting than the Ninth House section at the start, but everything is crumbling, drowning, broken, or decaying. There's quite a lot of body horror, grotesque monsters, and bloody fights. And the ending is not the best part of the book; roughly the last 15% of the novel is composed of two running fight scenes against a few practically unkillable and frankly not very interesting villains. I got exhausted by the fighting long before it was over, and the conclusion is essentially a series cliffhanger. There are also a few too many characters. The collection of characters and the interplay between the houses is one of the strengths of this book, but Muir sets up her story in a way that requires eighteen significant characters and makes the reader want to keep track of all of them. It took me about halfway through the book before I felt like I had my bearings and wasn't confusing one character for another or forgetting a whole group of characters. That said, most of the characters are great, and the story gains a lot from the interplay of their different approaches and mindsets. Palamedes Sextus's logical geekery, in particular, is a great counterpoint to the approaches of most of the other characters. The other interesting thing Muir does in this novel that I've not seen before, and that feels very modern, is to set the book in essentially an escape room. Locking a bunch of characters in a sprawling mansion until people start dying is an old fictional trope, but this one has puzzles, rewards, and a progressive physical structure that provides a lot of opportunities to motivate the characters and give them space to take wildly different problem-solving approaches. I liked this a lot, and I'm looking forward to seeing it in future books. This is not the best book I've read, but I thoroughly enjoyed it, despite some problems with the ending. I've already pre-ordered the sequel. Followed by Harrow the Ninth. Rating: 8 out of 10

26 April 2020

Russ Allbery: Review: The Last Goodbye

Review: The Last Goodbye, by Matt Potter
Publisher: Silvertail
Copyright: 2014-2016
Printing: 2016
ISBN: 1-909269-42-5
Format: Kindle
Pages: 308
In the contested space between the interested amateur and the trained expert lies the enthusiast. A topic has, for often inexplicable reasons, caught fire in their thoughts and become a mild obsession. They may not have formal training or systematic conceptual grounding beneath their interest, but they partly make up for that lack with sustained fascination. They research widely and obscurely about their particular focus, develop novel theories, see distinctions and classifications that few others would bother to investigate, and present their discoveries to anyone who will stand still long enough. And occasionally, when that interest happens to coincide with writing skill, they produce some surprisingly excellent essays or books. Matt Potter's enthusiasm is resignation letters.
Every damaging resignation letter, every cornered truth attack, every out-of-control speech by a former friend, is more than just an inconvenience, to be countered with positive spin and internal memos: it's an open challenge to the official version of the story, the perfectly controlled brand. They are breaks in an otherwise perfectly planned, smoothly executed narrative of the powerful. Holes in the program code. A rare, irresistible chance to hack into history's shiny, unstoppable operation.
The Last Goodbye: A History of the World in Resignation Letters is not, in truth, a history of the world. It is, first and foremost, a taxonomy, because there are types of resignation letters. The opening chapter, the truth bomb, is the type that one would expect upon discovering that someone wrote a book on the topic (that wasn't advice on how to write a good one). But there are other types, less heavy on the fireworks but just as fascinating. The unquotable expert construction. The knife in the back. The incoherent scream of rage. But also the surprisingly gentle and graceful conclusion.
It is the question that the letters themselves try in vain to answer, over and over again even as they explain, analyse, protest and bear witness to a million other details. The question is: Why? All the forces in the universe stack up against unburdening ourselves in a resignation letter. Professionally, it can be suicide. In practical terms, it is often self-defeating. Self-help books coach against unleashing its force; colleagues and confidantes urge caution, self-restraint. And yet we do it, and damn the consequences. We have no choice but to speak in sorrow, love, grief, cold anger, thirst for revenge, wounded pride, the pain of injustice, loyalty, pangs of regret, throes of vengeful madness, deluded righteousness, panic, black distress, isolation, ecstasies of martyrdom, and a million other shades of human extremity we need to say our piece even as we leave the stage.
The risk of the enthusiast's book is that the lack of structural grounding can leave the conclusions unsupported. A fair critique of this book is that it contains a lot of amateur sociology. Potter has a journalist's eye for motive and narrative, but some of his conclusions may not be warranted. But he compensates for the lack of rigor with, well, enthusiasm. Potter is fascinated by resignation letters and the insight they offer, and that fascination is irresistibly contagious. It's probably obvious that the chapters on truth bombs, fuck yous, and knives in the back have visceral appeal. The resignation letter as a force of truth-telling, as the revenge of a disregarded peon, as a crack in the alliance between the powerful that may let some truth shine through, is what got me to buy this book. And Potter doesn't disappoint; he quotes from both famous and nearly unknown examples, dissects the writing and the intent, and gives us a ringside seat to a few effective acts of revenge. That's not the best part of this book, though. The chapter that I will remember the longest is Potter's dissection of the constructed resignation letter. The carefully drafted public relations statement, the bland formality, the attempt to make a news story disappear. The conversation-ender. It's a truism that any area of human endeavor involves more expertise than those who have only observed it from the outside will realize, but I had never thought to apply that principle to the ghost-written resignation letter. The careful non-apology, the declaration that one has "become a distraction," the tell-tale phrasing of "spending more time with my family" is offensive in its bland dishonesty. But Potter shows that the blandness is expertly constructed to destroy quotability. Those statements are nearly impossible to remember or report on because they have been built that way: nouns carefully separated from verbs, all force dissipated by circuities and modifiers, and subtle grammatical errors introduced to discourage written news from including direct quotes. Potter's journalism background shines here because he can describe the effect on news reporting. He walks the reader through the construction to reveal that the writing is not incompetent but instead is skillfully bad in a way that causes one's attention to skitter off of it. The letter vanishes into its own vagueness. The goal is to smother the story in such mediocrity that it becomes forgettable. And it works. I've written several resignation letters of my own. Somewhat unusually, I've even sent several of them, although (as Potter says is typical) fewer than I've written. I've even written (and sent) the sort of resignation letter that every career advisor will say to never send. Potter's discussion of the motives and thought process behind those letters rang true for me. It's a raw and very human moment, one is never in as much control of it as one wishes, the cracks and emotions break through in the writing, and often those letters are trying to do far too many things at the same time. But it's also a moment in which one can say something important and have others listen, which can be weirdly challenging to do in the normal course of a job. Potter ends this book beautifully by looking at resignation letters that break or transcend the mold of most of the others he's examined: letters where the author seems to have found some peace and internal understanding and expresses that simply and straightforwardly. I found this surprisingly touching after the emotional roller-coaster of the rest of the book, and a lovely note on which to end. This is a great book. Potter has a good eye for language and the emotion encoded in it, a bracing preference for the common worker or employee over the manager or politician, and the skill to produce some memorable turns of phrase. Most importantly, he has the enthusiast's love of the topic. Even if you don't care about resignation letters going in, it will be hard to avoid some fascination with them by the end of this book. Recommended. This book was originally published as F*ck You and Goodbye in the UK. Rating: 8 out of 10

6 April 2020

Russ Allbery: Review: Thick

Review: Thick, by Tressie McMillan Cottom
Publisher: The New Press
Copyright: 2019
ISBN: 1-62097-437-1
Format: Kindle
Pages: 247
Tressie McMillan Cottom is an associate professor of sociology at Virginia Commonwealth University. I first became aware of her via retweets and recommendations from other people I follow on Twitter, and she is indeed one of the best writers on that site. Thick: And Other Essays is an essay collection focused primarily on how American culture treats black women. I will be honest here, in part because I think much of the regular audience for my book reviews is similar to me (white, well-off from working in tech, and leftist but privileged) and therefore may identify with my experience. This is the sort of book that I always want to read and then struggle to start because I find it intimidating. It received a huge amount of praise on release, including being named as a finalist for the National Book Award, and that praise focused on its incisiveness, its truth-telling, and its depth and complexity. Complex and incisive books about racism are often hard for me to read; they're painful, depressing, and infuriating, and I have to fight my tendency to come away from them feeling more cynical and despairing. (Despite loving his essays, I'm still procrastinating reading Ta-Nehisi Coates's books.) I want to learn and understand but am not good at doing anything with the information, so this reading can feel like homework. If that's also your reaction, read this book. I regret having waited as long as I did. Thick is still, at times, painful, depressing, and infuriating. It's also brilliantly written in a way that makes the knowledge being conveyed easier to absorb. Rather than a relentless onslaught of bearing witness (for which, I should stress, there is an important place), it is a scalpel. Each essay lays open the heart of a subject in a few deft strokes, points out important features that the reader has previously missed, and then steps aside, leaving you alone with your thoughts to come to terms with what you've just learned. I needed this book to be an essay collection, with each thought just long enough to have an impact and not so long that I became numb. It's the type of collection that demands a pause at the end of each essay, a moment of mental readjustment, and perhaps a paging back through the essay again to remember the sharpest points. The essays often start with seeds of the personal, drawing directly on McMillan Cottom's own life to wrap context around their point. In the first essay, "Thick," she uses advice given her younger self against writing too many first-person essays to talk about the writing form, its critics, and how the backlash against it has become part of systematic discrimination because black women are not allowed to write any other sort of authoritative essay. She then draws a distinction between her own writing and personal essays, not because she thinks less of that genre but because that genre does not work for her as a writer. The essays in Thick do this repeatedly. They appear to head in one direction, then deepen and shift with the added context of precise sociological analysis, defying predictability and reaching a more interesting conclusion than the reader had expected. And, despite those shifts, McMillan Cottom never lost me in a turn. This is a book that is not only comfortable with complexity and nuance, but helps the reader become comfortable with that complexity as well. The second essay, "In the Name of Beauty," is perhaps my favorite of the book. Its spark was backlash against an essay McMillan Cottom wrote about Miley Cyrus, but the topic of the essay wasn't what sparked the backlash.
What many black women were angry about was how I located myself in what I'd written. I said, blithely as a matter of observable fact, that I am unattractive. Because I am unattractive, the argument went, I have a particular kind of experience of beauty, race, racism, and interacting with what we might call the white gaze. I thought nothing of it at the time I was writing it, which is unusual. I can usually pinpoint what I have said, written, or done that will piss people off and which people will be pissed off. I missed this one entirely.
What follows is one of the best essays on the social construction of beauty I've ever read. It barely pauses at the typical discussion of unrealistic beauty standards as a feminist issue, instead diving directly into beauty as whiteness, distinguishing between beauty standards that change with generations and the more lasting rules that instead police the bounds between white and not white. McMillan Cottom then goes on to explain how beauty is a form of capital, a poor and problematic one but nonetheless one of the few forms of capital women have access to, and therefore why black women have fought to be included in beauty despite all of the problems with judging people by beauty standards. And the essay deepens from there into a trenchant critique of both capitalism and white feminism that is both precise and illuminating.
When I say that I am unattractive or ugly, I am not internalizing the dominant culture's assessment of me. I am naming what has been done to me. And signaling who did it. I am glad that doing so unsettles folks, including the many white women who wrote to me with impassioned cases for how beautiful I am. They offered me neoliberal self-help nonsense that borders on the religious. They need me to believe beauty is both achievable and individual, because the alternative makes them vulnerable.
I could go on. Every essay in this book deserves similar attention. I want to quote from all of them. These essays are about racism, feminism, capitalism, and economics, all at the same time. They're about power, and how it functions in society, and what it does to people. There is an essay about Obama that contains the most concise explanation for his appeal to white voters that I've read. There is a fascinating essay about the difference between ethnic black and black-black in U.S. culture. There is so much more.
We do not share much in the U.S. culture of individualism except our delusions about meritocracy. God help my people, but I can talk to hundreds of black folks who have been systematically separated from their money, citizenship, and personhood and hear at least eighty stories about how no one is to blame but themselves. That is not about black people being black but about people being American. That is what we do. If my work is about anything it is about making plain precisely how prestige, money, and power structure our so-called democratic institutions so that most of us will always fail.
I, like many other people in my profession, was always more comfortable with the technical and scientific classes in college. I liked math and equations and rules, dreaded essay courses, and struggled to engage with the mandatory humanities courses. Something that I'm still learning, two decades later, is the extent to which this was because the humanities are harder work than the sciences and I wasn't yet up to the challenge of learning them properly. The problems are messier and more fluid. The context required is broader. It's harder to be clear and precise. And disciplines like sociology deal with our everyday lived experience, which means that we all think we're entitled to an opinion. Books like this, which can offer me a hand up and a grounding in the intellectual rigor while simultaneously being engaging and easy to read, are a treasure. They help me fill in the gaps in my education and help me recognize and appreciate the depth of thought in disciplines that don't come as naturally to me. This book was homework, but the good kind, the kind that exposes gaps in my understanding, introduces topics I hadn't considered, and makes the time fly until I come up for air, awed and thinking hard. Highly recommended. Rating: 9 out of 10

25 August 2017

Dirk Eddelbuettel: BH 1.65.0-1

The BH package on CRAN was updated today to version 1.65.0. BH provides a sizeable portion of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages. This release upgrades the version of Boost to the rather new upstream version Boost 1.65.0 released earlier this week, and adds two new libraries: align and sort. I had started the upgrade process a few days ago under release 1.64.0. Rigorous checking of reverse dependencies showed that mvnfast needed a small change (which was trivial: just seeding the RNG prior to running tests), which Matteo did in no time with a fresh CRAN upload. rstan is needing a bit more work but should be ready real soon now and we are awaiting a new version. And once I switched to the just release Boost 1.65.0 it became apparent that Cyclops no longer needs its embedded copy of Boost iterator---and Marc already made that change with yet another fresh CRAN upload. It is a true pleasure to work in such a responsive and collaborative community.

Changes in version 1.65.0-1 (2017-08-24)
  • Upgraded to Boost 1.64 and then 1.65 installed directly from upstream source with several minor tweaks (as before)
  • Fourth tweak corrects a misplaced curly brace (see the Boost ublas GitHub repo and its issue #40)
  • Added Boost align (as requested in #32)
  • Added Boost sort (as requested in #35)
  • Added Boost multiprecision by fixing a script typo (as requested in #42)
  • Updated Travis CI support via newer run.sh

Via CRANberries, there is a diffstat report relative to the previous release. Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.